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ABSTRACT ^ . \ ^ 

Taking the ^position that the Classical Experimental 
Evaluation (GEt) Model does not do justice to the process of 
acquiring information necessary for decision making re planning, , 
programming, implementing, and recycling ^ogram activities,' this 
paper presents the Inductive, S ystem^ Process (ISP) evaluation model 
as an alternative to be used in evaluation of Title V rural 
development programs. Major components/of the ISP model are 
identified as: scenario negotiation ^mutual efforts to determine the 
purpose of the evaluation, search ^.u'fe program goals and criteria, and 
identify needed evid^ncaf for judgmeiit making) ; evidence collection 
(derived from the ConteyXt, Inputs, pVocess, ?rodiict Model, 
emphasizing techniques established via scenario negotiation); ^ 
judgment (description and evaluation of what occurred via the group, 
process approach, encompassing both summative and formative 
evaluation) • Specific Titl^ V evaluation is described as involving 
judgments about the organizational adequacy of the overall Title V 
delivery system and/or the attainment of individual Title V project 
goals/objectives in terms of required judgments, criteria, evidence, 
and evaluation procedures. Providing a systematic, framework for 
ansvering specific questions and structuring the e^luat ion process, 
the ISP model is presented as a guide for future Title V evaluations, 
(jc) _ 
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, \ PREFACiE 

\ 

Section 23.6(a) a£ Title V requires that each land grant university 
evaluate th^. impact? of its Title V program on the developi^nt of a specially 
selected target area- The evaluatJ.on of Title V- activities is especially 



challenging because of the unique nature of this Actn Federal administrators 
and Congress are interested in the ability of land grant universities to ^ 
bring about changes in rural areas through research (ijfiformation generation) 
and extension (information dissemination) activities. Land grant ur^iversities 




be^ng asked, therefore, to demonstrat^e whether they have the capability 

,/ . 

tou aid' in the development of rural areas b& xhey have in- the development of 

agriculture. The demonstration of such aid is a critical issue in future 

funding of the 'Act.. 

ti 

Bach of the states is charged with evaluating their own program efforts 
and with demonstrating their impacts. State administr^TorTS. have been given 
minimal res^ources for conducting programs and practically none for evaluating 
their efforts- The North Central Regional Center for Rural Development was 
brought into the Title V program to facilitate the program efforts of t\\Q 
several stfates i[f the region- The Center is prepared^ therefore, to help 
with the Title V program by providing furlrd^^ consultants, ideas^ and training 
required to successfully complete a Title V program in those states where 



assistance is needed . 
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Preparation of this report was supported through Title V funds from the 
North Central Regional Center for Rural Development and Is pne effort by the 
Center to assist state Tilcle V coordinators throughout the region. This 
report presents a model, which is still being developed .and modified, for 
evaluating the wide range of activities that- are part of Title V. Although 
the details of the model are still evolving > the model should^be helpfuX^aa^^ 
a tool chac raises certain issues about evaluation^ but not necessarily/as 
specifying evaluation techniques that provide aid the answer's/ Wes^'have tried 
to present some issues central to the evaluation process and wj^bln che 
contexc of Ti tie V progranuningi requirements . 

/ 



APPROACHES TO EVALUATION 

Inbroduction 

] 

reviewing numerous evaluation modelss Steele (1973) observed that 
much needs to be dOne in conceptualizing, ^(hodeling, and testing program 
evalCjation procedures. To this end ^ this paper will first explore two major, 
viewpoints on the purpose for evaluation, and a suggested model for evaluating 
development programs ^will be developed. Secondly the issues involved in- 
using the proposed model to evaluate an ongoing program will be reviewed and 
discussed..' llopefullyj through this process^ improvements will be made in the 
concGfptu^ilizitig J modeling, and testing of evaluation methods suitable for 
determining the impacts of development type programs. 

Limited resources available for social action programs, the growing 
emphasis of government agencies and citizen groups upon accountability ^ and 
llmita ti^jlS^ in the current development of evaluatlon^mo^ls all provide a 
rat/onale for the model to presented. 

The Convcn tional ITvalua tion Approach 

The conventional view of evaluation research defines evaluation as: 
1) prov.iding program admifilstrators with accurate Information about the 
consequences of their actUons (Caro^ 1969) ^ 2) providing the fact-finding 
procedures for discovering the results of planned social action (Hyman^ 
et al.t 1962) t and 3) ^ measuring the consequonc^d^'^o^ goal^oriented action 
(Criessmanj 1969).* These definitions of eva^iJation all focus thft evaltrStion 
process on goal-attainment. This ^ie^ of 'evalqation se.ek.6 to determine whether; 



or noc a program has accomplished its objectives^ i.e. > did Che "arrow hit 
the ^larget.*' It is commonly assumed in using this model that the "target'* 
(objectives) are clearly specified and unchanging, and that** the program 
"arrow'^ is unaffected by extraneous variables. Since the use of an exper- 
imental design facilitates control of such variables, evaluatfon research and 
the Classical Experimental Evaluation Model (CEE Model) often have been 
linked. . 

V 

Advantages and Use of the CEE Model 

/ The CEE Model has a number _of distinct and important advantages. These 
are similar to the advantages of the scientific method and include: 1) ob- 
jcctive and verifiable data, 2) causal inferences between progrtm inputs and 
products > arnd 3) generali^able results . Furthermore , the advantages of the 
CEE Model permit the elimination of factors other than the program being 
evaluated as causal explanation's of outcomes produced (Hyman> et al.> 1962),' {- 
and they eliminate Che tendency to confuse progress in marshalling inputs with 
the progress toward output targets (USAID> 1972) . 

In the CEE Model, as in all models, program development and evaluation 
planning must proceed together to insure- that properly measurable program 
objectives are developed and that an^ experimental evaluation design is 
feasible. Normally^ collection of data on goal attainment occurs before the 
program is implemented and after it is completed. Data may'be collected^ 
l^Owever , at different points during the operatj^pn of t)^e program. Wlicre this 
is done it Is intended to insure that changes disruptive to the experimental 
evaluation design do not occur. ' 



Obstacles co U sing the CEE "^Model ^ . . ^ 

— ■■ ^ ■ < 

Major obstacles in using the CEE Model generally Inciude: i) the 
difficulty (or impossibility) of effectively iraplemetiting the CEE Model in 
some field situations, 2) che^ need for judgments about the program that are 
not facilitated tliro^gli use of a tl£,E Model* 3) *the fact that program objectives 
may4^ be difficult to cLefine ^arui(or) opera tionalize , A) ethical problems in 
the selection and use of treatment and control gro^jps , 5)' lack of control by 
the researcher over Selectipn of persons into the prograrai 6) lack of access 
to program participants in the study population / and "7) ' difficulty in 
implementing necessary controls if the program already is underway- 
After examining the failure of one evfiluation effort chat used the CEE - * 
Model in an action prt^ram, Wcias and Rein (1969) concluded that: 

...When action programs' arc more like Mod^l City PlanrTing and less 
like innocuiation with a flu vaccine, an experimental model. for 

evaluating^ effectiveness is apt to be a mistake, ^ • . 

\ 

t - t 

They found tUikt the CEE effort failed because it dTd' n^Dt fit the" reality of 
the social-action program beingy^ evaluated. It coul<i x\bt be effectively ini- 
plement^d'^nor could it meet the informa tion 'needs <^ program administrators 
and decision makers. It would seem that large-scale^ social-ac^:ion programs 

" 4 

often may need to bo evaluated in ays different from those imposed by use. of 
atT experimental design. " ^ 

Need for Alte rnatives Foresh adowed 

If an evaluation is to be used In determining whether or not a program 
with clearly sf^ccified objectives should be continued, the use of the CEE 

\ 

Modei appears appropriate: The CEE Model does not appear to be appropriate, 
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however, for assisting program administrators and decision makers to manage ^ 
their activities. The CEE Model v^uld be difficult to u^e where the program 

Is new or innovative or where the objectives are' vague, ambigious, or evolving, 

I ... 

If the CEE Model Is not equipped to handle these situations, wh^t type of 
^valuation Is appropriate? 

An Alternative Evaluation -Approach 

We take the position that evaluation is not just assessment of goal- 
attainment buc also should include the process of acquiring information 
necessary for decision making about the planning, prog ramming ^ implementing, 
and recycling oC program activities. Evaluation should permit a probing of 
chamges in "targets'* during the course of program development and identify 
how effectively the "arrows" or programs are launched and ways they are 
affected In flight, * 

As a result of their work, Weiss and Rein (1969) argue for more qualitative 
and process-oriented evaluation research especially when action programs 
contain broad aims and assum<J nons tandarlzed forms^> They advocace a descrip- 
tive, inductive^ systems-proceSs approach to evaluation research. Tills 
approach focuses upon lc*arning whaD* is^ happening In che program (i>e,, wl^^t 
is being done) Wther than eyclLiaively focusing on what waa expected fo 
happen. Oeutscher {197A)*^ emphasizes that program^ must be carefully observed 
.CO determine whatf is- accually^ happening — not what proposals or program ob- 
jectives say is to hapt^cn. Thus^ riither than necessarily requiring clearly 
specified program objectives (and che -underlyiag theory of the program) from ^ 

program administrators Ijeforc evaluation can occur, the theor/ on wfrich the 

' ■ 

program l^basedt including explicit and impliclc objeo-civee, may have co be 

inductively discerned as part of the evaluatioi^ process, ^ 

9 - ; ' 

J'^ ^ ' * 



An Inductive appro'ach to "process analysis** (what is^ happening) Is a way 
to avoid a misplaced emphasis upon goal attainment and some. of the deficiencies 
of a CEE. Model approach\ This>pproacV recognize^ that the evaluator is 
involved in analysis of an outgoing social ^ct — one that is seen as in constant 
flux and amenable to new definitions * By assuming that things may^be changing 
during the course of a program, the research *ef fort shifts from assessing 
accomplishment of preordained goaj^s to the discCH^ery of **processual consequences" 
or to a conaideration of "wbat^ is happening" -(Deutsch^r, 197A) * 

Deficiencies in Current EvaJ."^^^^" Model Development . - 

As the view of evaluation has broadened and as deficiencies of the CEE^ 
Model are recognised ^ evaluation approaches have .proll fera ted . This prolifera-- 

■ \ ' ' ■ 

tion was triggered when established approaches to evaluation (primarily 
variants of the CEE Model) were judged inappropriate (Steele, 1975)-. There 
is a growing realiication that a design for projects with we|l--tlef Incd ob- 
jectives may be different than where objectives arc stated more generally or 
are emerging as the project utvfold^ (Brack, 1975)- The results of model 
building to date, however, *have not b^en entirely satisfactory* NuinerouQ 
approaches to evalua tioii* are .useful only when the ^valuator practices a form 
of *'prngmatic eclec-ticism'* in utilizing^theory. More than 50 different 
approaches to evaluation have been Identified. Most of these niv&luation. models ^ 
were designed only for specific field situations* Also., mo©t models define 

evaluation as concerned v^^th coii^ction oi data pertinent to program outputs 

J * ' + I 

and generally 'omit consideration of the prjoccssea by wl;iich Judgmeats are 

reached* One deficiency in these several approaches is tht failure of many 
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CO deal with value questions involved in evaluation efforcs (Steele^ 1973). ^ 
In other words, they have not explicitly identified the major values involved 
ilL_ma'klnfi/judgments nor have they provided procedures for doin^so. 

A generalized and unifying framework for the conduct of evaluation has 
noc emerged from tlvese diverse approaches to evaluation- These approaches 
have not led to a compreliensive framework that could be used to effectively 
guide the evaluation process. A recent exception to tliis generalization is 
work by Alkin and Fitz-'Gibbon (1.975), who developed a general framewj/rk for 
evaluation that serves to identify issues in the evaluation proces^. They 
have not, however, integrated this iseue evaluation back into a model that 
' treats data-c^illectlon^ needs, 

^ Directions aad Needs in DeveloplnR Evaluation Models • 

Steele (1973) notes four important .areas that need attention in the 
development of evaluation models. First, an evaluation model must be able 
to deal with real-life situations and to make contributions in an everyday 
^environment. Second^ there is Q need to increase objectivity^ but at tlie 
same time retain intimate contact with people involved in Che evaluation 
f>roceas. third, more at^Centloh should b,e given to crys,tallizj|^ig unstated 
assumptSirftns into firm procedures or guidelines. Fourth, mor^ attention should 
be paid to purposes^ rsttlonales^ overall approaclic^^ and to outcomes ^ 
SReciCically attributable' to the 'evaluation process. The model to be pre- 
sented i$ a beginning attempt at developing a- model th^t can meet these 
/ 

evaluation needs. It is > however > only in a , developmental stagfe at this time. 



OVE^tVIEW OF tH£ INDUCtlVE, SYSTEM^PROCESS 
EVAi,UATION/rSP) MODEL 

Uncroduction 

^ - DjSf ficulciGS wich the traditional CEE Model^^ new difeccioris being taken 

\ ^ ■ 

by the numerous non-CEE Models (and cheir shorccomlngs) > and the shifting 

\ 

definlcion of evaluation all'point up a need J^or t:he development of a model 
more appropriate for program evaluation. There is a real need fclr an alterrfl^ 
tive CO the CEE Model chat is more inductive, systems-oriented > ci>ncerned 

with processes a^^ well as outcomes, and has demonstrated field utility. 

\? 

f. 

An Initial outline for sucfi on evaluation model is described below and 
shown tn Diagram 1- AJ.chough lt;s several componerfts have received attention 
in the evaluation literature (e.g., especially Weiss ^and Rein, 1959; Steele^^ 
1973, 1975; Stuf-f lebeam^ 1967, 1968; Alkin , ' 1969 ; Moe, 197A; OeutschCr, 1974» 
1975; Alkin and Fitz-^Cibbon, 1975, eCcO> they have not been integrated into 
a comprtihensive model. The basic components of the ISP modeV "to be outii'hed 
inciiide : 1) nego tiating the secenar io, 2) evidence collectl#ni, and 3) judgments/ 
evaluation. Additionally, it Is important to understand tKe relation between 
evaluation and the phase in the development of the program being OvaJLuJtod. 

K ' 

1 , Negotia tinR the Scenario ' - " 

* ^ y 

Xn *'negot ting a scenario," the Gvaluacor must jolrv with varjCSmaj Iptere^t 

groups, prac t itionerg , ' and administtcitors in a mutual efFort to deterrnine the" 

1 

purpose(s) oE the evaluatlorf, to search out program goals and. reasonable "^-^ 

/ 

criteria for assessing tijiem, iind to identify the evidence necdc:d in making 
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1 ' . 

TOund judgments. This appVoach is described by Deutscher (l97A) in Xhe 

following manner; 

.-,It is desirable for the evaluator to begin to locate what pro- 
gram people are trying to do by watching them do it and listening 
.to them talk about it — in situ ! There is a cumulating body of 
evidence from sociolinguistics that if one listens to people talk . 
about what they are doing while they are doi-tig it> chances of 
understanding the activity are maximized. ' 

IP 

Pa^t of .the evaluator's task then is to discover what in fact is^ 
being attempted. After watching and listening he will begin to 
speculate' ^bout what is happening and can then begin to engage - 
practitioners or administrators in a dialogue in an effort to 
negotiate the reality of the situation. ^ 

Initial negotiations will directly mold the evaluation process and 
affecd everything that follows. Negotiation will identify the type of in- 
formation that should be collected, how it should be analyzed, and how it 
should be reported. Once negotiation of a scenario is complete, the evaluator 
can move forward in designing and Implementing data callecti9n efforts. 

Despite the^ 4,mportance placed on negotiating -d^'scenario, systematic 
procedures and gui^Ielines for accomplishing it have not been developed* 

II. Evidence Collection 

The next stage in the ISP approach, evidence collection. Is derived 
from the CIPP Evaluation Model (Context, Inputs, Processes, -Product) 
developed by Stufflebeam (1967, 1968) which was modif iedi by Moe C197A) ^ and 
from the seven-level hierarchy of evidence suggested by Bennett and Nelson 
(1975)- Negotiation of a scenacla^nd the phase -of program development are 
important precussors of the specific types of data to be collected. The 
CIPP data collection format serves to guide and strupture data collection 
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efforts by suggesting types of evidence that should be obtained^ but it does 

not necessarily sjjecify th^ techniques to be used in their coflec^tion. 

Techniques appropriate for use in evidence collection efforts should be 

derived from the negotiation of the scenario^ not the reverse, ^ 

" The. Vevidence collection" phase should involve evidence related to; 

! , 1) * Defining the cont^t within which the program is to be pursued^ 
including., establishing some base lines or known starting points 
from whicn^lanned attempts to achieve goals and objectives can ^ 
be measured. This includes A^^P^ts > activities^ and people that 
are involved in tW^^. program^ or involved with the problem in tne 
pre-planning phase, Et. might also include reactions to the pre- 
planning st^ate of afjfairs., 

2) Documenting the inputs > the things^, that are done» the programs 
and activities initiated and the reaoujrces used to bring about 
change and to achieve goals and objectiv^es.. This might appro- 

^ ' priately include documentation of resources activities, and 
■ people Involved, 

3) Documenting the processes or the ways in which programs .are 
implemented. It would detail ways in which planning is done» 
the decisions made> the communication channels established^ ^ 
the interaction patterns that . emerge > the critical i^ncidents 
that occur> interpretations of and changes in policies^ and 
other features. The same elements and considerations mentioned 
for documenting the inputs may be involved. Additionally > data 
may be collected on reactions^ changes in knowledge > ^titudes > 
skills and aspirations (KASA changes). 

A) Documenting the outputs or the outcomes (effects) of what is 
done. This may appropriately include consideration of inputs,> 
activities > people involvement > reactions > KASA changes > practice 
changes^ and ultimate results of practice changes. 

The CIPP elements provide guidance as to the format and focus of data 

collection efforts. The model does not customarily require use of an 

experimental evaluation design. Instead^ it serves to isolate important 



considerations r'elated to the program development process and identifies 
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types of evidence appropriate for- particular data collection efforts, 

\ 

Although it may not be possible,- pr necessary, to have a complete set of 

\ 

data (evidence) fc>r each of the four CIPP elements identified above, the 
CIPP format helps in \developing an evaluation that is comprehensive and 
flexible. Finally, the CIPP evidence collection format is flexible enough 
for Use where goals and objectives are not clearly specified or are evolving. 
The model identifies types of evidence to be collected and. facilitiates an « 
inductive approach to evaluation, which makes it more likely th'af unanticipated 
conse(^uences of programs will be discerned. 

III , Judgments/ Evaluation ^ / - 

Once the data has been collected and summarized, the process of setting 
"value" on the data begins. This phase is central to the evaluation. There 
are" two /major facets io examining the results of a program — description and 
evaluation . (Steele , 1975). Description provides evidence of' what occurred. 
Evaluation involves makiiig judgments as to the-adequacy of what occurred. 
Judgments are improved when they are made by comparing evidence (data . 
collected) about the aspect to be judged against criteria of what shouJ^d 
exist, or what is valued. Evidence and criteria help in forming sound 
judgments, but neither constitute judgment- 
Community and human resource development are value laden concepts 
(Beal, 1974). Thus, all development programs and their evaluation are 
intimately tied to the values of program participants or '"stakeholder" 
groups. Among the issues to be resolved are **Who will make the necessary 
judgments or evaluation of a development program?'* and "How will these 
judgments be madef'* 

15 
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Several alternatives are available in making j udgments about program 
outcomes (summatlve evaluation) . Advocates who adhere strictly to the 
classica'l experimental model of evaluation (e.g., Suchman* 1967', Ferman , 
1969; Freeman and Sherwood, 1965; Longest 1975 ; Campbell, 1971) argue that 
conclusions a^out goal attainment ^are strengthened when statistical com*^ 
parisons show tHat the programs did, or did not, accomplish their stated- 
objectives. This approa-ch cannot be used, however, wftere objectives are 
evolving or a CEE Model cannot be implemented. * ^ 

A more realistic alternative is sugge^ed in Logsdon's (1975) "Group 
Process Model", of evaluation. Logsdon proposes that judgments (evaluations) 
be made in group (discussions invc>lving researchers » program administrators, 
and participants. They should collectiyely review program goals* processes* 
and evidence related to t^he prograig^ oper^t^gn and impacts, problems or 
failures along with successes^ and recommendations should be J-dentif led , and 
value(s) placed on the program and its activities. Through this "group- 
p.^oce3s" approach to evaluation* a summary document including data and 
evaluation can be developed for use J>y administrators and decision makers in 
planning future programs. The group involved in this process should be 
fairly small (5-7 persons)- toHncrease manageability and to facilitate inter 
action. The group process evaluati9n technique is consistent with the 
philosophy advanced by other students of evaluation (e.g.* Tripodi, et; al.* 
1971; Steele, 1973, 1975). 

Evaluation also may occur before the progrart is completed, and the final 
results known. Interim or formative evaluation is designed to provide 
immediate feedback for the purpose of altering the ongoing operation of 
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programs. The focus is usually upon Inputs and processes as evidence of 

I ' / 

expecDQd impacts or results. Evaluation of inputs or processes' is no 



made In terms pf their relevance for result? desired or expected. This 



rmally 
is 

iot ln> t\ 



evaluation is usually an intuitive process; flpal results are hot ln> they 
remain unknown. Formative evaluation of this nature cannot be handled 
within the format of^ the CEE Model. Yet; a purpose of many evaluations is to 
facilitate management and development of ongoing programs. Standardization - 
of procedures for formative evaluations^ however^ is ^tlll needed. These 
procedures may be similar to those discussed for summative evaluation. 



Program Development and Relation to Evaluation 

In order td effectively use the XSP model, as described above, it must 
be placed in the proper program development context. Both Situf f lebeam (1967)^ 
and Bennett and Nelson (1975) note the relationship between evaluation 
processes and program development. Three phases in the program development 
proiiess may be identified — planning^ programmlng/speclf JL^atlon > and Imple^ 
mentation. The planning phase is basically concerned with *'what to do.* 
Here the concern is with identifying priority problems and ultimate objectives. 
Programmlng/speclf icatlon Is the program develdpment phase designed to 
determine how to utilize resources to meet program goals and objectives. 
This is ^ concern with "how to do It/* In this phase shorter-term "enabling*' 
objectives are selected. Responsibility for reaching these objectives is 
accepted by new or existing specialized organizations > and staff ^or these 
organizatio^ns^ls recruited and trained. The. Implementation phase of program 
development relate^ to "doing It.** This means actually conducting the 

- \ 
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program^ including contacting additional people to participate in it. The 
ultimate aims of ^the program are achieved through, implementation. 

In evaluating any program, it is important to identify wMch phase of 
the program I3 being evaluated and which phase the program is in when the 
evaluation occurs. This provides clarification of the program issues and 
concerns that, can be appropriately dealt with in the evaluation since the 
program has a different focus for each phase. Also, the emphasis of the 
program phase provides directions for evidence collection. Clearly, any 
evaluation effort must be placed in, aud be operationali^ea within, the 
context of one of the thr^l^e program development phases. The general outline 
of the process involved in ^aluattng each program development phase, however, 
need not vary from one phase to^the next. Evaluation effortp should, ideally, 
be developed simultaneously with the program, Not only would, a fuller eval- 

r 

uation of all stages of the pr6gram development be possible, but the quality 
would be enhanced as well. 

The Reality of the Evaluation Context 

The reality of the evaluation situation is typically less than ideal, 
especially with mQst large-scale federally supported development programs , 
Development programs (or many of their individual projects and activities) 
typically have progressed through several of the initial program development * 
phases before any serious attention is given to evaluation. ^ Where the need 
(or requirements) exists for ''local" participation and coordination with 
Other institutions and agencies, program development that is less than ideal 
tends to occur , frequently , Program planning in "such cases usually takes 
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place In a very short time period. Thus, the development of objectives an 

strategies is necessarily general or vague. Initial conceifn Is with getting 

the program going (e.g., funded and implemented). it is only '^,£ter program 

y 

implementation has begun that attention Is turned to evaluation. The re- 
sulting negotiations that occur will tend to be rather limited. The same is 
true for the data collection efforts. Adequate attention and data collection 
efforts often cannot be given to all four elements of the CIPP format. An 
evaluation can, however^ still be conducted even though It may be limited. 
Limitations will be clear^ but following the ISP approach will still allow 
the best possible evaluation within the existing circumstances t and it Is 
better than no attempt at systematic evaluation. Evaluation situations that 
diverge from the ^'ideal" cannot be ignored by the social scientist. 

TITLE V EVALUATION 

introduction^ 

The I^P model previously discussed is being developed and applied in 
the evaluation of^ rural development activities ifi Iowa sponsored under Tltltf 
V o£ the Rural Development Act of 1972. We feel the proposed ISP model can 
be used In developing an evaluation process for many different types of 
development programs. The purpose of this section Is to develop and present 
some of the major issues and concerns Involved In the evaluation of a state- 
level Title V program* These Issues and concerns for Title V relate to five 
of ^ the six elements in negotiating the scenario and could provide a beginning 
point for negotiating the evaluation scenario in different types of development 
programs . 

10 



Overview of Title V 




ThG purpose o£ Title as specified in Che Rural foeveloVment ^ct of 
1972^ is to support progr'ams or rural development so as "to encourage and 
foster a balanced national development that provides opportunities for ^ 
Increased nuinbers oC Americans to work and enjoy a high quality of life 
througjiout the nation,..," The primary objectives of Title V are;, 

to encourage and foster a balanced national development that 
provides opportunities for increased numbers of Americans to 
work and enjoy a high quality of life dispersed throughout our 
nation by providing the essential programs of rural development; 

2) to provide chose ] Involved with public services and invest- 
ments in rural areas or chat provide or ^^y provide employment 
in these areas the best available scientific^ technical^ 
economic ^ organ izat ional ► environmental ► and management inf or- 
mation and knowledge useful to them, and to assist and encourage 
them> In the interpretation and application of this Information 
to practical problems and needs in rural development; 

3) to provide research and investigations in all fields that liave - 
as their purpose the develof>ment of useful knowledge and infor- 
mation to assist those planning^ carrying out ^ managing^ or 
investing in facilities ^ services ^ business , or other enter- 
prises ^ pubii*^and private,- that may contribute to rural 
development ; 

■ ' ■ ■ . ■ 

A) to enhance the capabilities of colleges and universities to 
perform the vital public service roles of research, transfer, 
and practical application of knowledge in support of rural 
development; and 

5) to expand research on Innovative approaches to small farm 
management and technology and extend training and technical 
assistance to small farmers so that they may fully ^utilize 
the best knowledge on sound economic approaches to small 
farm operations* (This last objective has not been funded 
by the Congress' , ) 

Witliln these objectives ^ it is clearly stated that their attainment rests 
upon a program for '*rural development/* What Is "rural development"? What 



20 



activities and projects of the TitljJ;^-.^l^gram are consistent with "rural 

clevc^lopment /' and should therefore be-coirtriButing to attainment of the 

above Objectives? Aa defined in Title rural development includes: 

1) the planning, financing^ and development of facilities and 
services in rural areas that contribute to making these areas 
desirable places in which to live and make private and busit\ess 



investments 



r 



2) the planning, development^ the expansion of business and 
Industry in raral areas to provide *l*n^:Teas(e^d employment and 
income ; ^ 

3) the planning^ development, conservation, and use of land, 
water, and other n'atural ^sources of rural areas to maintain, or 
or enhance the quality of the environment for people and 
busin^iss in rural areas ; and 

the processes and procedures that have said objectives as their 
major purposes. 

This specification of objectives and Activities that are appropriate 
for inclusion in a staters Title V program can be used as a guide for 
developing and(or) examining th^ Title V progr^ and its related activities. 
Title V also states that each state's program "...must include research and 
extension activities directed toward identification of programs which are 
likely to have the greatest impact upon accomplishing the objectives of rural 
development in both the short and longer cerm/^ This suggests that while 
'each state's immcdinte Title V objectives may differ somewhat from those 
objectives stated in flDA 1972, they should be consistent with the larger 
Title V objtictives and be auch that the state objectives will contribute 
or lead to attainment of the larger objectives of Title V. 



2 I. 
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Purpose of Tl/tile V Evalua'tilon ^ ^ 

Evaluation of the Title V program and Its related activities Is ongoing. 
Section 23, 6X^) of the Tide V Regulations requires that each state provide 
*for evaluation of the impact of its program activities upon rural development, 

I 

The general \}ut-fi'o^^f ,^ll Title V evaluation efforts Is to aid decision 
makers and admlnist;rfi6prs at federal^ scate^ and local levels l^i designing, 
admlnl^irefriag^, and cftrtauctlng current and future rural development programs, 

^(Judgmej3Lfc«^esired wlch respect to Title V and the criteria -suggested for 
use from the" federal level imply Interest In a two-level evaluation* One 
level of the evaluation (organizational evaluation) centet^^^ upaii^ makin^^ ^ ~* 
judgments about the organizational adequacy of the overall Title V delivery 
system. The second level of evaluation (project evaluation) Is to focus upon 
the attainment of the goals and objectives of Individual Title V activities- 
Diagram 2 ^shows the relation and emphasis, of the required evaluations 
for each program development phase for Title V. - Each cell represents an 
evaluation that could be conducted- Wlien evaluation is not included in the 
development of Title V activitl^^& and because the Tit-le V program is ^tlll 
in process, evaluation must focus Of>\ tlie'^rogrammlng/specification and 
implementation cells* When the individual Title V projects are in- different 
stages of development, the fuii evaluation model cannot be applied to all' 
projects* The intent, however, is to evaluate each project for as many stages 
as possible . 

Organl^fl tlonal EvaluaLlbn * Judgments Required ^ 

With respect to the overall Title V program^ officials at the federal 
level have identified three elements of the overall organization of the Title 
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V program about which judgments are required. These are:' 1) the organizational 
adequacy of the delivery s'yatetn, 2) organizational involvements (kind and 
extent), and 3) ^the nature of relationships between organizations (utilization 
o'f resources). The^^tter two elemer^ts are really -p rocesses involved in 
shaping the organisation of the Title V delivery system,- The major focus, 
then, is upon the organizational adequacy and opera-tion of the Title V 
delivery system. This emphasis is consistent with the fourth objective of 
^ritle V stated in the Rural Development Act of 1972. 

Organic tlonai^VaXuat ion : Criteria " , 

The basic judgments to be made in t^e organizational evaluation relate 
to \he organJ-zational adequacy and operation of the delivery system. ^ Federal 
level administrators ask .that these judgments be made in terms of the 
undGfiiicd criteria of "usefulness" and "effectiveness*" What this, means is 
that sotue conceptualization of "usefulness" and "effectiveness'* must be agreed 
upon by the vario.us participants in the 'Title V delivery system. This will 
require - dialogue and In te rac tion between va^rious groups to negotiate 
acceptable criteria for evaluating the "usefulness'' and(or) "effectiveness" 
for the organization of the Title V delivery system. Some possible criteria 
for evaluating the organizational adequacy of the Title V delivery system 
might 'be : ^ 

1) ' Significance - Whether 'O'r not the organization of the delivery 

system is worthwhile when conditions (inputs, situation, context, 
etcO* are considered, 

2) Effectiveness in terms of intent - The degree to^which the 
organization of the delivery system approicimates the intent 
of state level objectives. 
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3) Effectiveness i n terms of mission ee to which 

developed delivery system contributes to^the mission of Title V* 

A) Responsiveness - The degree to which the delivery system is^ 
meeting the needs of those it Is serving*. 

5) Equity - The degree to which the delivery siftptera is-raeeting the 
needd of selected groups or clients raorfe than of others. 



Ocganizational Evaluation i Evidence 

Data"" collection efforts for the organizational evaluation should center 
upon variables relating to;- 1) the organization ot the Title V delivery 
system and the involvements and relationships related t^o it> and 2) the 
criteria developed for use in the evaluation* The major source o^ data may 
be the actors in the Title V delivery systetn* These actors in the Title V 
delivery system ^Dight include: 1) pcojecc dicectors for each niajor individual 
.project funded^ 2) members of thescate Rural Development Advisory Council^ 
3) membets of any local Rural Development Advisocy Council > A) local elected 

ficials t 5) non-Title agency or ocganiza tional personnel involved in 
acciviciee contributing to Title V rucal development activities^ and 6) 
members of the local extension scaff. 

Data collection may occur chrough a number of techniques* For an 
ort^ing dcvelopmenc program^ data collection, efforts mlghc at)propriately 
include: 

1) a survey (questionnaire and(or) interview) of the .maj orWctors of 
the Title V system^ 

2) monitoring procedures, whereby Title V personnel provide certain 
information to the evaluator on a. periodic basis^ and 

3) the evaloator^s observations of ongoing activities and efforts* 

/ 

4 

/ 

21 




\ 



20 



It should be noted that monitoring procedures^ and the evaluator s observations 
may be equally applicable to project data collectiop efforts^^ 

Q^rfianizational. Evaluation: Procedures for Making Judgments ^ 

An evaluation committee composed of representatives of varioiis interests 
involved In Title V may be used to further complete the ''negotiation of the 
scenario/' This same committee aiso may function In the making oJf^final 
JudgmenD^ about the Title V organization after data collection is Complete, 
The actual organization, composition, and operation of such an evaluation 
committee in roaking judgments about the organizational adequacy of Title V 
needs to be negotiated before evalua t ion ' efforts b^gin. .^^ 

Project {^valuations : Judgments Required 

Wliereas the organizational evaluation focuses upon the organizational 
adequacy of the delivery system, the evaluation of individual"- projects 
focuses upon judgments about the attainment of project goals. The emphasis 
upon proj^t goals and obj^^^lves follows Erom a concern at the federal level 
with: 1) evaluating the ptrs^ess toward achieving objectives stated in 
the Annual Plans of Work, and 2) determining the degree to wfiich specific 
needs and problems have been Identified , addressed , and affected . 

Proj ect Evaluation : Criteria 

Tlie minimum basic criterion to be used in evaluating individual projects 
should relate to the attainment of project goals. Other criteria may be 
added where appropriate by project directors, Title V administrators, or 
the evaluator. Some other po^^lble criteria may be: 
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1) Adequacy of p'erformance - when the performafr^e Is compared to 
the total need. 

2) Impact - the strength of the project an()(o'r) activity Influence 
upon exposed, individuals * 

3) Significance - whether or not the- results produced are worthwhile 
when conditions (inputs, situation> context, etc>) are considerftcL, 

A) Effectivep^s in terroa of intent - the degree to which performance 
approxiniates the intent of the project aTid(or) activity goals*^ 
and objectives. 

5) Effectiveness in terms of mission - the extent to which the project 
-activities contribute to the mission of Title V (rural develop^ 
merrt — goals and objectives related to) - 

6) Keaponaiveness ^ whether or not the project is^meetlng the needs 
of those it is serving. 

7) Kqulty whether or not the ptoject is meeting the needs of some 
groups or clients more than others. ' 

'* 

Proj ect Evaluations ; Evidence 

The evidence to be used in evaluating Individual Title V projects 
necessarily will vary from project to project. The <ividence should be ^ ^ 
appropriate to the goals of the individual project. Although some data 
may be collected for certain individual projects through the data collection 
procedures outlined for the organizational evaluation> additional data 
specifically related to each project's activities may need to be collected. 
This may not be' equally possible or feasible in all cases. Development of 
data collection efforts for projects rests with the ovaluator, but must be 
coorxllnated and negotiated with the individual^ro j ect director to insure that 
project activities arc not disrupted or adversely affected. 



11 



The Title V Gvaluator should have major rcsponsiblity for constructing 

V ^ 

evaluation data collection Instruments and initial summarization and analysis 
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of data collected. Project personnel > hdwever^ may be responsible in some 
cases for actually collecting the data as a part of their project operation. ' 
The project personnel also, may be responsible for identifying participants > 
and should assist the evaluator with construction of any evaluation instruments 
to be used. 

In short> the conduct of project evaluations should be a cooperative 

It 

activity between the evaluator and the project director^ Evaluation becomes 
more meaningful when enriched by the knowledge and understanding that project 
personnel have of their activities and clients. 

Project Evaluations: Procedures for Making Judgments 

As in making judgments for the organizational evaluation^ judgments 

/ 

about individual projects "may be^ made by an 'jevaluation committee. The same 

/ ^ / ^ 

evaluation committee could be used> or a /eparate committee forpied for each 

^ project to be"^ evaluated 

/■ 

GUIDING AND IMPLEMENTING THE EVALUATION i 
,/ / PROCESS: SUMMARY /conclusion 

/' 

7 

* I^y^order to satisfactorily develop and implement the evaluation process 
for Title V Rural Development programs^ a number of major questions or 
issues must be addressed. These are: 

1) What phases of the Title V program can be evaluated? 

2) What is the purpose qf £he evaluation? 

3) What is the role of the evaluator? What is his relation to 
program administrators? 

^) What crucial judgments must be made to complete the purpose? * - 
How will thesel judgment^ be made? By whom will, these judgments 
be made? 
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5) What criteria are germane to those Judgments? 

6) What type of evidence is needed? How "p^re" must It he? 

7) What are the most efficient sources ^of such evidence? 

8) How are the dataf collected and the Judgments made to be reported? 
The negotiation of answers for these question^ will structure the actual 
conduct of the evaluation and the data collection activities related to it • 
Once the data collection efforts have been completed, the following questions 
must be addressed in order to complete the evaluation process. 

1) ^ How does the evidence compare with the criteria? 

2) What are the resultant Judgments? 

3) What do the judgments mean in terms of 'the purposes of the 
evaluation? 

4) What will recommendations or decisions based on those Judgments 
mean to those involved? How is the evaluation and(or) 
recommendations to be reported? ^ ^ 

The ISP Model provides a systematic framework ^?^or answeirii^g these 



questions and structuring the evaluation process. Although specific pro 




cedures and techniques used may di-^fer from state to^tate<^ the^^gineral issues 
and guiding principles for the evaluation are the same. These must be 
addressed systematically and aa/a whole. Their eff^c^t upon other must 

be explicitly understood. 
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INDUCTIVE SYSTEMS -PROCESS EVALUATION FRAMEWORK (ISP MODEL) 



. IV. RECYCLING of 
the Program and/or 
Activi ties 


> 





Decisions/^ 

Re c omme nda ti o ns 



Waking Judgments! 

T 



Determining value 
and meaning of 
data 



I, NEGOTIATING THE SCENARIO 



purpose of the 
Evaluation 



Role of the Evaluator 



^Judgments Required 



t Criteria 



Evidence 



procedures For 
Making Judgments 



II, Evidence Collection 



Defining the^ Context 



Documenting inputs ^ 
and Processes 



Documenting the Results 



Summari zation 
and/or Analysis 



III, JUDGMENTS/teVALUATION 

(SUMMATIVE EVALUATION) 



Kevisions in xne 
Ongoing Program 
and/or Activities 



Decisions/ 
Recommendations 
?c 



Making Judgments 





Determining Value 

and Wean ing of Data 





JUDGMENTS/teVALUATION 
(FORMATIVE EVALUATION) 



Diagram 1 
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RELATION OF TITLE V PROGRAffl DHVELOPf£E?lT FmSSS 
TO TH2 -EVALUATION OF TITLE V 



1. Planninf^ 



II . Programming^ 
Specification 



CO 

O 



III • Implementation 



Title V Organizational Evaluation Title V project Evaluation 



(1) Federal objectives --operation 
of the Title V system 



(n) State Title V objectives 



(1) Organization of Title V 

delivery system to meet 

state Ob jective (inputs 
and processes } 



(n) Title^ V organizational 
system '( results } 



{1^ Operation of the- organisati'onal 
delivery system through time 
(inputs and processes) 



(n) Changes in delivery system and/ 
or its operation (results — 
"effectiveness and usefulness") 



( 1 ) Federal objectives-- 

what is to be accomplished 
in rural development by 
the system 



( n) Project objectives 



( 1) Organizational development 
of project activities and 
efforts to meet project 
objectives ( inputs and ' 
processes ) 



(n) Project organization 
operating procedures 
('results ) 



and 



(1) OperatioTf of project 
organization and 
activities through time 
( inputs and ' processes ) 



(n) Impact of project activities 
and attainment of objectives- 
Title V and project (results) 
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